Testing

The Remote Protocol has unit- and functional tests located under different folders:

Marionette: remote/marionette/.
Shared Modules: remote/shared/
WebDriver BiDi: remote/webdriver-bidi/

You may want to run all the tests under a particular subfolder locally like this:

% ./mach test remote

Unit tests

Because tests are run in parallel and xpcshell itself is quite chatty, it can sometimes be useful to run the tests in sequence:

% ./mach xpcshell-test --sequential remote/shared/test/xpcshell

The unit tests will appear as part of the X (for xpcshell) jobs on Treeherder.

Browser Chrome Mochitests

We also have a set of functional browser-chrome mochitests located under several components, ie. remote/shared/messagehandler/test/browser:

% ./mach mochitest remote/shared/messagehandler/test/browser/browser_*

The functional tests will appear under the M (for mochitest) category in the remote jobs on Treeherder.

As the functional tests will sporadically pop up new Firefox application windows, a helpful tip is to run them in headless mode:

% ./mach mochitest --headless remote/shared/messagehandler/test/browser

The --headless flag is equivalent to setting the MOZ_HEADLESS environment variable. You can additionally use MOZ_HEADLESS_WIDTH and MOZ_HEADLESS_HEIGHT to control the dimensions of the virtual display.

The add_task() function used for writing asynchronous tests is replaced to provide some additional test setup and teardown useful for writing tests against the Remote Agent and the targets.

Puppeteer tests

In addition to our own Firefox-specific tests, we run the upstream Puppeteer test suite against our implementation to track progress towards achieving full Puppeteer support in Firefox when using the WebDriver BiDi protocol. The tests are written in the behavior-driven testing framework Mocha.

Puppeteer tests are vendored under remote/test/puppeteer/ and are run locally like this:

% ./mach puppeteer-test

You can also run them against Chrome as:

% ./mach puppeteer-test --product=chrome

By default the mach command will automatically install Puppeteer but that’s only needed for the very first time, or when a new Puppeteer release has been vendored in. To skip the install step use the --no-install option.

To run only some specific tests from the whole test suite the appropriate test files have to be updated first. To select specific tests or test groups within a file define exclusive tests by adding the .only suffix like it.only() or describe.only().

It is also possible, similar to how it works in CI, to run tests in chunks:

% ./mach puppeteer-test --this-chunk=1 --total-chunks=2

More customizations for Mocha can be found in its own documentation.

Test expectation metadata is collected in remote/test/puppeteer-expected.json via log parsing and a custom Mocha reporter under remote/test/puppeteer/json-mocha-reporter.js

Check the upstream Puppeteer test suite documentation for instructions on how to skip tests, run only one test or a subsuite of tests.

Testing on Try

To schedule all the Remote Protocol tests on try, you can use the remote-protocol try preset:

% ./mach try --preset remote-protocol

But you can also schedule tests by selecting relevant jobs yourself:

% ./mach try fuzzy